Automate publication process #58
Conversation
|
This is so awesome!!! I've been dreaming about something like this. If I have time I'll test it out later today. This won't actually make the readme file for you though, right? I was thinking that could be a nice thing to do too, likely as a separate command - generate the nice table view of the values from the converter or fiboa schema in some way. |
fiboa_cli/publish.py
Outdated
| if not os.path.exists(pm_file): | ||
| log(f"ogr2ogr geo.json", "info") | ||
| exc(f"ogr2ogr -t_srs EPSG:4326 geo.json {parquet_file}") | ||
|
|
||
| log(f"Running tippecanoe", "info") | ||
| exc(f"tippecanoe -zg --projection=EPSG:4326 -o {pm_file} -l {dataset} geo.json --drop-densest-as-needed") |
There was a problem hiding this comment.
This has some heavy dependencies that would need to be installed somehow. Running into GDAL installation issues everytime, I feel like this might be hard to solve properly...
There was a problem hiding this comment.
Maybe have these ones be a little more 'opt-in'? Like we just check if the libraries are installed, and if not it just says 'please install GDAL and tippecanoe', and skips the steps.
There was a problem hiding this comment.
Thanks for the valid points in the review. I realized this pull-request was a work-in-progress that "works on my machine", but wanted to check if it was an interesting feature for the project.
To get a data-set published/updated, you need to follow the HOWTO and you will need gdal & tippecanoe on your machine. I relied on os.system() calls as opposed to python-libraries to avoid introducing more dependencies. Whenever I see a list of detailed instructions like in the HOWTO, I'm always afraid I (or one of my project-members) will mess up / forget something. A script like this -with enough guidance and sanity checks- will keep us all safe. Let's see if I can get some more sanity checks in there...
There was a problem hiding this comment.
I think it's a good idea. I think this might need to be a bit more interactive, giving instructions to users and waiting for confirmation, ... - more like a step-by-step wizard ;-)
There was a problem hiding this comment.
Ok. @m-mohr I'm not sure what part you're considering making interactive. We could pre-generate the README and LICENSE by scraping the specific page from the data-survey. Or we could request a copy/paste-input of secrets from source coop.
Currently the script stops with some instructions (write README, install tippecanoe, copy the export AWS_ commands in the terminal.
There was a problem hiding this comment.
I think he meant just meant interactive with the requirements for the 'additional things' - like prompting someone 'would you like to generate pmtiles y/n?', and then have it check if tippecanoe is installed, and if not tell the user 'tippecanoe must be installed to generate pmtiles, do XYZ to install it'.
ivorbosloper
force-pushed
the
publish_cli
branch
from
e5da422
to
4c8097c
Compare
Example:
fiboa publish fi -e finland -c ~/Downloads fi_dataAutomates https://github.com/fiboa/data/blob/main/HOWTO.md#each-time-you-update-the-dataset step 9-19